A framework for automatic semantic video annotation

The rapidly increasing quantity of publicly available videos has driven research into developing automatic tools for indexing, rating, searching and retrieval. Textual semantic representations, such as tagging, labelling and annotation, are often important factors in the process of indexing any video, because of their user-friendly way of representing the semantics appropriate for search and retrieval. Ideally, this annotation should be inspired by the human cognitive way of perceiving and of describing videos. The difference between the low-level visual contents and the corresponding human perception is referred to as the ‘semantic gap’. Tackling this gap is even harder in the case of unconstrained videos, mainly due to the lack of any previous information about the analyzed video on the one hand, and the huge amount of generic knowledge required on the other. This paper introduces a framework for the Automatic Semantic Annotation of unconstrained videos. The proposed framework utilizes two non-domain-specific layers: low-level visual similarity matching, and an annotation analysis that employs commonsense knowledgebases. Commonsense ontology is created by incorporating multiple-structured semantic relationships. Experiments and black-box tests are carried out on standard video databases for action recognition and video information retrieval. White-box tests examine the performance of the individual intermediate layers of the framework, and the evaluation of the results and the statistical analysis show that integrating visual similarity matching with commonsense semantic relationships provides an effective approach to automated video annotation

Helminth-induced Th2 cell dysfunction is distinct from exhaustion and is maintained in the absence of antigen

T cell-intrinsic regulation, such as anergy, adaptive tolerance and exhaustion, is central to immune regulation. In contrast to Type 1 and Type 17 settings, knowledge of the intrinsic fate and function of Th2 cells in chronic Type 2 immune responses is lacking. We previously showed that Th2 cells develop a PD-1/PD-L2-dependent intrinsically hypo-responsive phenotype during infection with the filarial nematode Litomosoides sigmodontis, denoted by impaired functionality and parasite killing. This study aimed to elucidate the transcriptional changes underlying Th2 cell-intrinsic hypo-responsiveness, and whether it represents a unique and stable state of Th2 cell differentiation. We demonstrated that intrinsically hypo-responsive Th2 cells isolated from L. sigmodontis infected mice stably retained their dysfunctional Th2 phenotype upon transfer to naïve recipients, and had a divergent transcriptional profile to classical Th2 cells isolated prior to hypo-responsiveness and from mice exposed to acute Type 2 stimuli. Hypo-responsive Th2 cells displayed a distinct transcriptional profile to exhausted CD4+ T cells, but upregulated Blimp-1 and the anergy/regulatory-associated transcription factors Egr2 and c-Maf, and shared characteristics with tolerised T cells. Hypo-responsive Th2 cells increased mRNA expression of the soluble regulatory factors Fgl2, Cd38, Spp1, Areg, Metrnl, Lgals3, and Csf1, and a subset developed a T-bet+IFN-γ+ Th2/Th1 hybrid phenotype, indicating that they were not functionally inert. Contrasting with their lost ability to produce Th2 cytokines, hypo-responsive Th2 cells gained IL-21 production and IL-21R blockade enhanced resistance to L. sigmodontis. IL-21R blockade also increased the proportion of CD19+PNA+ germinal centre B cells and serum levels of parasite specific IgG1. This indicates a novel regulatory role for IL-21 during filarial infection, both in controlling protection and B cell responses. Thus, Th2 cell-intrinsic hypo-responsiveness is a distinct and stable state of Th2 cell differentiation associated with a switch from a classically active IL-4+IL-5+ Th2 phenotype, to a non-classical dysfunctional and potentially regulatory IL-21+Egr2+c-Maf+Blimp-1+IL-4loIL-5loT-bet+IFN-γ+ Th2 phenotype. This divergence towards alternate Th2 phenotypes during chronicity has broad implications for the outcomes and treatment of chronic Type 2-related infections and diseases

Optimal Dominant Motion Estimation using Adaptive Search of Transformation Space

The extraction of a parametric global motion from a motion field is a task with several applications in video processing. We present two probabilistic formulations of the problem and carry out optimization using the RAST algorithm, a geometric matching method novel to motion estimation in video. RAST uses an exhaustive and adaptive search of transformation space and thus gives -- in contrast to local sampling optimization techniques used in the past -- a globally optimal solution. Among other applications, our framework can thus be used as a source of ground truth for benchmarking motion estimation algorithms. Our main contributions are: first, the novel combination of a state-of- the-art MAP criterion for dominant motion estimation with a search procedure that guarantees global optimality. Second, experimental re- sults that illustrate the superior performance of our approach on synthetic flow fields as well as real-world video streams. Third, a significant speedup of the search achieved by extending the mod el with an additional smoothness prior

Multimedia in forensics, security, and intelligence

With the proliferation of multimedia data, it has become necessary to secure this content from illegal use, efficiently detect and reconstruct illegal activities from it, and use it as a source of intelligence. Serious challenges arise from the sheer data volume, however. The multimedia research community has developed many exciting solutions for dealing with video footage, images, audio, and other multimedia content over recent years, including knowledge extraction, automatic categorization, and indexing. Although this work forms an excellent foundation for protecting and analyzing multimedia content, challenges remain in the complexity of the targeted material, the lack of structure and metadata, and other application-specific constraints. This special issue provides an overview of current research following this mission. The articles originally appeared at the ACM Multimedia 2010 Workshop on Multimedia in Forensics, Security, and Intelligence (MiFor). The six high-quality contributions cover various approaches in the field, ranging from the visual recognition of faces and tattoos to the discovery of near duplicates and content tampering

Topic models for semantics-preserving video compression

Most state-of-the-art systems for content-based video understanding tasks require video content to be represented as collections of many low-level descriptors, e.g. as histograms of the color, texture or motion in local image regions. In order to preserve as much of the information contained in the original video as possible, these representations are typically high-dimensional, which conflicts with the aim for compact descriptors that would allow better efficiency and lower storage requirements. In this paper, we address the problem of semantic compression of video, i.e. the reduction of low-level descriptors to a small number of dimensions while preserving most of the semantic information. For this, we adapt topic models - which have previously been used as compact representations of still images - to take into account the temporal structure of a video, as well as multi-modal components such as motion information. Experiments on a large-scale collection of YouTube videos show that we can achieve a compression ratio of 20 : 1 compared to ordinary histogram representations and at least 2 : 1 compared to other dimensionality reduction techniques without significant loss of prediction accuracy. Also, improvements are demonstrated for our video-specific extensions modeling temporal structure and multiple modalities

Ontology-Assisted Object Detection: Towards the Automatic Learning with Internet

We focus on the automated analysis of spectator crowd, that is, people watching sport contests alive (in stadiums, amphitheaters etc.), or, more generally, people \u201cwatching the activities of an event [\u2026] interested in watching something specific that they came to see\u201d [2]. This scenario differs substantially from the typical crowd analysis setting (e.g. pedestrians): here the dynamics of humans is more constrained, due to the architectural environments in which they are situated; people are expected to stay in a fixed location most of the time, limiting their activities to applaud, support/heckle the players or discuss with the neighbors. In this paper, we start facing this challenge by following a social signal processing approach, which grounds computer vision techniques in social theories. More specifically, leveraging on social theories describing expressive bodily conduct, we will show how, by using computer vision techniques, it is possible to distinguish fan groups belonging to different teams by automatically detecting their liveliness in different moments of the match, even when they are merged in the stands. Moreover, we will show how, only by automatically detecting crowd\u2019s motions on the stands, it is possible to single out the most salient events of the match, like goals, fouls or shots on goal